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ABSTRACT 



Based on a decade of evaluation experience, the Center for 
Substance Abuse Treatment (CSAT) has developed the Integrated Evaluation 
Methods (lEM) Package, a series of conceptual and methodological applications 
to enhance CSAT-funded evaluation activities. Products in the lEM Package are 
organized within an evaluation framework constructed on the basis of 
accumulated experiences among evaluation professionals. The framework is 
based upon evaluation strategies, structures, and approaches appropriate for 
substance abuse treatment evaluators and providers . This concept paper 
presents state-of-the-art conceptual models addressing issues related to 
coordination of treatment and evaluation activities, and integration of 
clinical, performance, and evaluation information. Specifically it summarizes 
lessons learned from CSAT evaluation activities, identifies evaluation 
concepts and methods necessary to address the lessons learned, and describes 
an overarching evaluation framework and integrated evaluation approach to 
provide a foundation for future evaluation planning activities. Appendix A is 
"Integrated Evaluation Methods Package: A Guide for Substance Abuse Treatment 
Knowledge -Generating Activities --Executive Summary"; Appendix B is "Editor's 
Note"; Appendix C is "List of Expert Panel Members"; and Appendix D is 
"Systems Evaluation Bibliography." (Contains 10 figures, 2 tables, and 30 
references . ) (MKA) 
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Foreword 



Over the last 10 years the Center for Substance Abuse Treatment (CSAT) has 
accumulated a great deal of experience in substance abuse treatment evaluation implemented 
through coordinating centers, cross-site efforts, and national studies. The importance and value 
of integrating ongoing evaluation activity into a system for treating substance abuse problems is 
widely recognized. Also widely recognized, however, is that current evaluation-generated 
knowledge and practice are often under-utilized, due in part to the lack of an integrated approach 
to capturing information with which to measure and improve treatment effectiveness, efficiency, 
and performance. CSAT recognizes that such an integrated evaluation approach will more 
effectively support current and future knowledge-generating activities. 

Based on a decade of evaluation experience, CSAT has developed the Integrated 
Evaluation Methods (lEM) Package, a series of conceptual and methodological applications, 
including concept papers, technical assistance materials, and analytic tools, to enhance CSAT- 
funded evaluation activities. Products in the lEM Package are organized within an evaluation 
framework constructed on the basis of accumulated experiences among internationally known 
treatment service evaluation professionals. Thus, the framework is based upon evaluation 
strategies, structures and approaches appropriate for substance abuse treatment evaluators and 
providers. The framework follows a standard set of evaluation activities: planning, selecting a 
design, developing data requirements and collection instruments, collecting and analyzing the 
data, and reporting the evaluation findings. (A summary description of the lEM Package is 
contained in Appendix A to this document.) 

This concept paper and its companion documents, Self-Adjusting Treatment Evaluation 
Model; Building Team Capability to Fully Implement and Utilize the Self-Adjusting Treatment 
Evaluation Model; Adding “Value” to CSAT Demonstrations', Performance Measurement for 
Substance Abuse Treatment Services, and Client Levels of Functioning as a Component of 
Substance Abuse Treatment Services Evaluation present state-of-the-art conceptual models 
addressing issues related to coordination of treatment and evaluation activities, and integration of 
clinical, performance and evaluation information. Specifically, this concept paper summarizes 
lessons learned from CSAT evaluation activities; identifies evaluation concepts and methods 
necessary to address the lessons learned; and describes an overarching evaluation framework and 
integrated evaluation approach to provide a foundation for future evaluation planning activities. 

Sharon Bishop 
Project Director 
NEDTAC 
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I. Introduction 



This document describes an integrated evaluation methods (lEM) approach and its 
genesis. Therefore, this document summarizes CSAT evaluation activities and the lessons 
learned from these activities, identifies the evaluation concepts and methods necessary to address 
the evaluation lessons learned, and provides an evaluation framework as the foundation for the 
integrated evaluation methods. The CSAT evaluation framework was constructed on the basis of 
accumulated experiences among nationally and internationally known treatment service 
evaluation professionals. This paper (and the lEM package) reflects and incorporates these 
experiences, gained over the past decade. 

1. DEVELOPMENT OF THE INTEGRATED EVALUATION METHODOLOGY 

DOCUMENT 

Since its inception, the Substance Abuse and Mental Health Services Administration 
(SAMHSA) Center for Substance Abuse Treatment (CSAT) has provided Federal leadership to 
improve substance abuse treatment accessibility, quality, effectiveness, and efficiency. CSAT’s 
mission and associated activities have evolved from a services support orientation to a 
knowledge-generating and performance assessment orientation. This evolution is evident in the 
current SAMHSA policy on evaluation as well as the CSAT approach to evaluation and 
embodied in the lEM (Marsh et al., 1996). 

CSAT initially (FY 1990-91) focused on enhancing treatment services directly and 
employed national evaluations to determine treatment efficacy and impact. The national 
evaluations used a sample of CSAT-fimded treatment providers and clients which precluded the 
generalizability of precise evaluation information to all treatment approaches and populations. 
Likewise, the national evaluations did not allow for analyses that addressed the treatment 
services delivered by each treatment provider. 

CSAT then funded local process (FY 1992) and outcomes (FY 1993) evaluations at 
CSAT-funded treatment sites so as to increase the utility of the evaluation information for local 
treatment service providers. Local evaluations proved insufficient to fully assess these sites. 
Furthermore, local evaluation findings could not be generalized at the national level. Therefore, 
in FY 1995, CSAT initiated multi-/cross-site evaluations, in addition to local process and 
outcome evaluations, for CSAT treatment providers in order to provide both program level and 
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site-specific evaluation findings.' Because of the diversity among CSAT programs and among 
the treatment approaches used in the substance abuse treatment field, the cross-site approach also 
was insufficient to address national treatment issues. In FY 1 996, the CSAT focus shifted again 
to knowledge development and application (KD&A) activities. The KD&As develop and test 
innovative exemplary treatment approaches and practices wherein the primary grant-funded 
activities are clinical trials and national evaluations rather than service provision. 

The national, cross-site, and local evaluation experiences provided CSAT with numerous 
opportimities to acquire experience about different approaches to evaluating CSAT programs, 
various evaluation methods to assess substance abuse treatment services, and an array of 
evaluation implementation strategies within the substance abuse treatment milieu. These 
evaluation lessons identified the need for: 

■ Increased coordination and collaboration within and among all participating 
organizations 

■ More systematic planning for knowledge-generating activities 

■ Clearer specifications in knowledge-generating activities announcements and 
statements of work 

■ Integrated evaluation methods and tools 

■ Measures, data and data collection procedures and tools, applicable to all CSAT 
knowledge-generating efforts 

■ A comprehensive evaluation strategy that involves methods development and testing, 
national evaluation studies, program evaluations, multi-site and cross-site analyses, 
and local process and outcome evaluations 

■ Analytic strategies across KD&A study sites within a program and across KD&A 
programs. 

These evaluation lessons also provided CSAT with the opportunity to develop a series of 
conceptual and methodological applications to enhance and coordinate CSAT-fimded evaluation 
activities. CSAT now recognizes a need for integrated evaluation methodologies, which 
incorporate these conceptual and methodological applications, to support current and future 



CSAT supports both multi-site and cross-site evaluations. For an operational definition of these two 
approaches, see Section 1.3: Multi-site/Cross-site Evaluations. 
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operationalized is presented in this paper. 
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2. HOW THIS PAPER IS ORGANIZED 

The underlying developmental process for the integrated evaluation methods approach is 
graphically depicted in Exhibit I-l. As indicated, the CSAT evaluations identified lessons 
learned which provided the impetus for adapting existing and developing new evaluation 
methods and tools. The evaluation methods and tools constitute an evaluation framework which 
provides the foundation for an integrated evaluation methods package. 

This document contains five sections: 

■ The Introduction which provides background information on the development of the 
CSAT approach to substance abuse treatment evaluation 

■ A summary of the various evaluation approaches CSAT has used over the last 
decade including national, local, and cross-site/multi-site evaluations and knowledge- 
generating activities 

■ Lessons learned from these various evaluation approaches which provide insights 
and knowledge about applying evaluation methods within the substance abuse 
treatment environment 

■ A substance abuse treatment evaluation framework which provides a logical 
structure for the analytic concepts developed in response to lessons learned from 
previous CSAT evaluation approaches 

■ Integrated evaluation methods approach which describes the evaluation methods, 
procedures and tools and their interrelationships. 

The collection of evaluation concepts, methods, and tools described in this document has been 
developed to support CSAT-funded national substance abuse treatment evaluations and 
knowledge-generating activities. The integration of these components within an integrated 
evaluation methods package is designed to ensure that future evaluations are structured to 
support SAMHSA knowledge-generating goals and policies. Appendix A contains a summary 
description of the lEM Package. Documents comprising the lEM Package are being made 
available at the Caliber Associates NEDS contract web site at http://neds.calib.com. 
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Exhibit I-l 

Development of Integrated evaluation Methods Package 




II. CSAT Evaluation Activities 

Evaluation is the systematic application of social research procedures to assess the 
design, implementation, and utility of social intervention programs; in other words, evaluators 
use social research methodologies to judge and improve the ways in which human service 
policies are developed and human service programs are conducted (Rossi, Freeman, 1993). 
Hence, evaluation is used at both the policy and treatment operations levels. 

CSAT recognizes the value of evaluation for the treatment improvement and 
identification and adoption of exemplary practices mission as demonstrated by the fact that 
CSAT treatment service demonstrations have always included evaluations, although the 
approach to evaluation has evolved over time. Between FY 1990 and FY 1998, the differences 
in CSAT evaluations have been more in the structure, focus, and scope of the evaluation rather 
than in the presence or absence of the evaluation. 

Between FY 1990 and 1998, CSAT used four approaches to evaluating substance abuse 
treatment services: 

■ National evaluations which collect provider and client information from a sample of 
grant funded local implementation sites across several CSAT program areas 

■ Local evaluations which focus on the specific grant activity at the treatment 
implementation site 

■ Multi-/cross-site evaluations which include all local implementation sites within one 
CSAT program area 

■ Knowledge development and application (KD«&;A) activities which rigorously 
examine various treatment models and approximate traditional clinical trials. 

The local evaluations and the multi-/cross-site evaluations involved significant funding of 
treatment services, with lesser funding levels for the evaluation component. The national 
evaluations and KD«&;As involve significant funding of the evaluation activities, with limited 
funding of direct services. 

best copy available 
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No single approach provided all of the information needed to evaluate exemplary 
practices and improve substance abuse treatment. Even so, each of these evaluation approaches 
has provided knowledge about the performance of the CSAT-funded activities which are 
operating within local communities, and about the impact of those activities on the services 
offered and the clients served. Each of these evaluation approaches also has provided knowledge 
about the strengths and limitations of specific evaluation strategies and about the factors that 
enhance and the factors that impede evaluation implementation. 

Since the focus of this discussion is on the evaluation process rather than on the 
evaluation results, lessons learned about how to successfully conduct CSAT evaluations are most 
germane. By strengthening the evaluation process, CSAT will enhance future evaluation results 
and be better able to answer with more certainty such questions as: What types of substance 
abuse treatment are most effective for whom and at what cost? Therefore, the following 
paragraphs briefly describe the CSAT evaluation approaches listed above so as to provide a 
context for the lessons learned from these evaluation approaches. 

1. NATIONAL EVALUATIONS 

The initial CSAT evaluation approach was to fund local treatment services and to 
separately fund a national treatment outcome study. The National Treatment Improvement 
Evaluation Study (NTIES) was a Congressionally-mandated five-year study designed to answer 
questions about the effectiveness of treatment services supported by CSAT. To demonstrate the 
value of comprehensive treatment, CSAT established three major demonstration program areas 
and made 157 multi-year treatment enhancement awards across 47 states and several territorial 
areas during FY 1 990-1991 . The recipients of these awards were not a representative sample of 
all treatment units, since they focused on serving large cities, minority populations, and other 
special populations (e.g., criminal justice populations). 

NTIES was designed to answer the following questions about the treatment services 
supported by CSAT; 

■ What treatment improvements were deployed? What is the evidence that improved 
treatment services yielded effective results? 

■ How were CSAT funds used? How many and what kind of treatment services and 
clients were affected by the CSAT-fiinded activities? 
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NTIES used a two-level study design. The first level focused on the service aspects of treatment 
including treatment orientation, number of clients served, budget, staffing distribution, and 
specific use of CSAT funds in all service delivery units (SDUs) eligible to receive CSAT grant 
support. The second level focused on client outcomes and collected data at treatment entry, 
treatment exit and approximately 12 months following treatment exit for all clients enrolled in 10 
percent of the level one SDUs. Details of the methodology and evaluation results are provided 
in the full NTIES report (NORC, 1997). 

The national evaluation of the CSAT FY 1990 and 1991 programs has provided 
important information about the provision of substance abuse treatment, including treatment 
modalities, staffing, and services. NTIES also provided a very comprehensive data set about 
clients who entered treatment in the FY 1990-91 time frame and about the impact of substance 
abuse treatment on these clients’ behaviors 12 months following treatment exit. The SDU data 
and client demographic and outcomes data are still vital to addressing substance abuse treatment 
effectiveness questions about specific populations and treatment services that were included in 
the NTIES database. NTIES data collection ended in 1995. 

The utility of the NTIES data had three major limitations, however. First, a large national 
evaluation with longitudinal follow-up data collection, by definition, requires an extensive 
amount of time. For CSAT, the seven years (1990 - 1997) needed for NTIES development, 
implementation, completion, analysis, and reporting precluded addressing treatment service 
performance and outcome questions during the operation of the activities included in the 
evaluation. Second, because of the NTIES sampling of SDUs and clients for the in-depth data 
collection, the national evaluation approach had limited operational utility for the local substance 
abuse treatment providers. There were insufficient resources to do treatment site specific 
analysis. Third, new knowledge-generating activities initiated by CSAT after NTIES was 
undertaken (those initiated in F Y 1 992 through F Y 1 996) could not be incorporated within 
NTIES and still maintain the study’s integrity. 

CSAT also funded special emphasis evaluations including The Evaluation of Job Corps 
Drug Treatment Enrichment Project (DTEP), The D.C. Superior Court Intervention Program, and 
D.C. Initiative. DTEP was designed to provided CSAT and the Office of Job Corps with 
information about the costs and benefits of providing substance abuse treatment services to a 
residential program for adolescents. The D.C. Superior Court Intervention Program was a unique 
random assignment study that looked at drug treatment services provided to a criminal justice 
population. In addition, the study provided a cost-benefit analysis for three categories of 
beneficiaries: the criminal justice system, the public, and the defendant. The D.C. Initiative 
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conducted a follow-up study to an experiment designed to test the efficacy of providing enhanced 
or standard therapeutic community treatment. The study compared clients from two treatment 
approaches on drug use, criminal activity, and employment status and addressed limitations of 
prior research through random assignment, higher follow-up rates, and objective measures of 
drug use and criminal history. 

These special emphasis evaluations provided advantages and limitations that were similar 
to the NTIES. Namely, these evaluations identified knowledge about the relative effectiveness 
of substance abuse treatment and/or treatment approaches for distinct populations and these 
evaluations accumulated comprehensive data sets about treatment and comparison cohorts. 
Limitations included insufficient timeliness for policy development (due to up-ffont design 
efforts and follow-up data collection) and limited utility for local sites participating in the 
evaluations (particularly, the DTEP study). 

2. LOCAL EVALUATIONS 

To address the limitations of the national evaluation approach, CSAT supported local 
process evaluations beginning in FY 1991 and local process and outcomes evaluations beginning 
in FY 1992. The intention was to supplement the large national evaluation with more in-depth 
evaluations focused on a specific SDU within an individual community or provider. These local 
evaluations have varied widely within CSAT program areas as well as across program areas, 
however. Differences are associated with the amount of resources allocated for local evaluations 
as well as with the objectives and types of the local evaluations. 

The amount of resources allocated to the local evaluations has had an impact on the 
evaluation comprehensiveness and quality. Resource levels determined the extent to which local 
evaluations are able to: (1) comprehensively collect information about the substance abuse 
treatment services and the clients; (2) incorporate control or comparison groups within the 
evaluation design; and (3) collect follow-up data to support outcome analyses. 

The types of local evaluations and the local evaluation designs within and across CSAT 
program areas have varied and have included one or more of the following: 

■ Process evaluations which measured treatment service implementation within local 
sites 
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■ Client based evaluations which monitored client flows and services received, 
primarily using pre- post- designs and no comparison or control group 

■ Outcome evaluations using quasi-experimental designs which included comparison 
groups, follow-up data collection and the analysis of treatment outcomes and impacts 

■ Intensive local research efforts which pooled multiple research dollars (including 
CSAT grant dollars) to test specific hypotheses about specialized treatment 
approaches and/or distinctive populations (e.g., psycho-social development of infants 
within a CSAT Pregnant and Postpartum Women’s program). 

Some treatment sites performed process evaluations but no outcome evaluations. Some 
treatment sites performed outcome evaluations without doing a process evaluation, which made 
interpreting the outcome data less informative than desired. 

In general, the local evaluations provided site-specific information about substance abuse 
treatment implementation, treatment clients and client flows, and, in some cases, substance abuse 
treatment effectiveness. Local substance abuse treatment implementation and the organizational 
and administrative barriers to full implementation were documented, to varying degrees, by 
many of the local process evaluations. Few local evaluations fiilly reported demographic 
characteristics, treatment experiences, treatment exit status (e.g., completion of treatment plan), 
and planned versus actual lengths of stay for all clients treated by the evaluated SDU. Local 
evaluations that used an experimental or quasi-experimental design, including follow-up data 
collection, reported treatment outcomes and treatment effectiveness findings. There were very 
few local evaluations that were sufficiently resourced for this level of evaluation rigor, however. 

CSAT has been working to combine the findings from local evaluations so as to better 
determine the overall effectiveness of specific CSAT demonstration programs as well as the 
overall effectiveness of substance abuse treatment services. The value of the local evaluation 
findings for these purposes, however, has been limited. Essentially, knowledge gained from 
local evaluations could not be combined, meaningfully, to address program, policy, or research 
questions at the national level. Local evaluation findings could not be synthesized due to the 
variations among local evaluation objectives, designs, methods, measures, time frames, data 
collection points, sample sizes, analyses performed, and report structure. In addition, local 
evaluation findings did not generalize to the total populations within a given geographic location 
or CSAT program area and, therefore, were not informative outside of the local context. 
Consequently, CSAT continually pursues efforts to generate program evaluation information 
which is more global in terms of its application and interpretation. 
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3. MULTI-SITE/CROSS-SITE EVALUATIONS 

CSAT has employed multi-site and cross-site evaluation strategies to assess program- 
wide implementation, treatment outcomes, and costs. A review of the methodological literature 
and professional discussions among NIDA researchers suggested that the terms “multi-site” and 
“cross-site” are used inter-changeably within the evaluation field and that differences between 
these approaches are associated with the operational definitions. Due to the analytic purposes of 
the cross-site approach, CSAT operationalized the cross-site evaluation as a prospective 
evaluation which analyzed common sets of data collected across multiple local sites using the 
same treatment protocol, to answer national evaluation questions. Multi-site evaluation methods 
were used for CSAT programs which used different treatment protocols within one or more 
treatment sites. 

Each multi-site/cross-site evaluation included a local evaliration which focused on 
collecting and analyzing in-depth treatment services data and client data. Local evaluations also 
addressed specific treatment service characteristics and/or client populations which are unique to 
the locale. Cross-site evaluation activities included: developing a common set of measirres at 
referral, intake, in-treatment, treatment exit, and follow-up; data on the cost of providing the 
treatment services; developing a common database; and analyzing process and outcome variables 
to determine whether statistically and clinically significant treatment outcomes and impacts are 
realized at the CSAT program level. 

The multi-site/cross-site evaluation approach, by design, attempted to combine the 
advantages of the in-depth local evaluations with the advantages of a program-level evaluation 
by generating comparable local evaluation data which could then be pooled to answer national 
evaluation questions at the CSAT program area level. Implicit in this approach, however, are the 
same limitations experienced by earlier CSAT national and local evaluations. Specific 
limitations included: 

■ Prolonged design and plarming phases, post-award, due to limited cross-site 
specifications in funding armouncements 

■ Unanticipated reluctance among geographically and programmatically diverse CSAT 
treatment providers to embrace the systematic multi-site/cross-site evaluation 
approach 
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■ Unanticipated barriers to collecting comparable SDU and client data from disparate 
local SDU operations and local evaluators who are using different data sources, data 
collection instruments, and data collection reference points 

■ Unanticipated challenges to standardizing data quality control procedures among 
multiple data collectors across data collection sites 

■ Insufficient planning for and implementing follow-up data collection requirements 
resulting in additional delays to obtaining follow-up data. 

In short, the CSAT multi-site/cross-site evaluations are generating knowledge about the CSAT- 
funded activities included in the cross-site efforts. The multi-site/cross-site evaluation approach, 
however, has not sufficiently addressed CSAT’s need for timely or comprehensive knowledge 
across program areas which is directly applicable to improving the provision of substance abuse 
treatment services. 

4. KNOWLEDGE-GENERATING ACTIVITIES 

Technically, all CSAT evaluation activities currently support knowledge development 
objectives and, to the extent that this knowledge is useful to other programs, these evaluations 
also support knowledge application activities. Beginning in FY 1996, the CSAT KD&A 
mission, however, reflects specifically a shift in the CSAT role away from supporting the 
provision of direct services with a small evaluation component to one of supporting the rigorous 
evaluation of treatment models to include clinical trials. 

While there are differences across the new KD&As (FY 1996 and beyond), these 
programs have similar characteristics, including: 

■ Pre-designed study specifications which guide the entire effort 

■ Study sites which provide the clinical interventions or local data collection support or 
both and, thus far, coordinating centers that conduct multi-site/cross-site activities 

■ Steering committees which include representatives from the Federal agencies, 
coordinating centers, and study sites. 

Several of the KD&As are designed to accommodate specific limitations of the multi-site/cross- 
site evaluation approach in that the standardization of the evaluation measures, data, and data 
collection activities within a program area are specifically incorporated within the terms and 
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conditions of the grants. The challenges of combining data across cross-site activities remains, 
due to the absence of core data specifications across CSAT KD&A program areas. In the 
absence of an overarching approach to evaluation across the KD&A efforts, the ability to 
compile data within national data sets that is sufficient to address the multitude of analytic issues 
that characterize substance abuse treatment evaluation will be limited, if even possible. Other 
lessons that have been learned from the accumulation of CSAT treatment evaluation experiences, 
not directly addressed by the KD&A structure, are described below. 
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III. Lessons Learned 



Over the last decade, CSAT has made a significant investment in generating knowledge 
about the effectiveness and efficiency of substance abuse treatment through its evaluations and 
KD&A efforts. Rather than providing the results of these evaluations, however, the purpose here 
is to summarize the lessons learned from designing and implementing these evaluations. 

Several types of evaluation lessons predominate in the CSAT experiences, thus far, 
including the need to: 

■ Coordinate the roles and activities of the evaluation sponsors, evaluators and service 
providers including the linkage of treatment evaluation with clinical practices 

■ Design evaluations which assess all treatment components and provide detailed study 
goals, evaluation design requirements, sample size specifications, data points and data 
requirements, as well as expected evaluation products prior to grant awards 

■ Ensure rigorous and consistent provider assessments through coordinated, 
comprehensive evaluations which accommodate all outcomes and cost analysis 
requirements 

■ Provide an overarching approach that facilitates the integration of data across 
evaluation activities and program areas. 

The following paragraphs summarize types of lessons learned from CSAT evaluations so as to 
provide a foundation for future evaluation planning activities. 

1. NEED FOR COORDINATION 

The field of substance abuse treatment has been evolving over the past 30 years, and 
today, substance abuse treatment services are extremely diverse with intermingled philosophies 
and modalities. During this time period, the evaluation of substance abuse treatment has 
employed traditional research models whereby the evaluator has been separate from the 
provision of treatment services. The CSAT evaluation philosophy however has evolved to 
address treatment services diversity and to encourage the integration of evaluation and treatment 
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operations. CSAT evaluations, by definition, involve multiple organizational units and 
numerous professional staff. Coordination among Federal sponsors and evaluation activities and 
the coordination of evaluation findings and treatment provider operations is essential. Lessons 
learned about functional and subject matter coordination are mentioned below. 

Increasing coordination for evaluation planning and implementation. Historically, 
CSAT-funded treatment and evaluation activities have not been conducted using a fully 
integrated process, since each organizational function contains highly specialized expertise with 
distinct roles and responsibilities. For knowledge-generating activities, however, close 
coordination of treatment services provision and evaluation planning and implementation greatly 
enhances the efficiency of these activities. Specifically, the coordinated specification of 
treatment theories and the components which address these theories (such as designs, data 
collection and reporting, and timeframes) increases the overall potential yield of these efforts. 
Coordination enhances the integration of the treatment approaches and the data collected about 
the interventions, across efforts. The integration of treatment, evaluation and data collection 
activities across organizational entities supports broad-based analyses which, in turn, strengthens 
the quality and credibility of the overall planning, implementation, and evaluation efforts (Patton, 
1987). 



Increasing linkages between treatment evaluation and clinical practice. Knowledge 
about substance abuse treatment has grown rapidly through research and program evaluations but 
the incorporation of this knowledge into treatment services has not. The knowledge-generating 
activities are designed to address this issue and ensure ongoing knowledge application. The 
integration of research and evaluation findings within the planning, implementation, and 
operations of substance abuse treatment is critical to ensuring that newly developed theories, 
newly tested methods, and documented exemplary treatment practices are available for the 
benefit of the clients served and those who manage or provide those services (National Institute 
on Alcohol Abuse and Alcoholism, 1991; National Institute on Drug Abuse, 1998). 

2. NEED FOR COMPREHENSIVE AND COMPARABLE EVALUATION 
STRATEGIES 

Given the complex, multi-faceted nature of substance abuse, substance dependency, and 
substance abuse treatment, substantial variation exists within and across treatment settings, and 
among philosophies and modalities. Yet, few research and evaluation studies precisely define 
the treatment domain, and even fewer compile evidence of the unique and/or interactive effects 
of these variables on clients. Moreover, the complexity of treatment processes coupled with the 
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complexities of human behavior complicate further attempts to identify and measure specific 
treatment components and their results. Measures of treatment philosophy, modality, intensity, 
services, and staff characteristics must be considered in relation to population characteristics and 
dependency diagnoses (Institute of Medicine, 1 990). 

In addition, substance abuse treatment evaluations must be coordinated and, whenever 
possible, integrated. Given the complexities in treatment and dependency patterns, it is 
necessary to rigorously and systematically evaluate problem- and population-specific 
interventions. Innovative treatment approaches are usually (given costs and other resources) 
limited to a finite population in any given effort. But, since scientific evidence of substance 
abuse treatment effects requires numerous observations, it is important to increase the number of 
observations (clients) so as to extend the generalizability of individual treatment evaluations. 
Increasing comparability among data sets across study sites and program areas, and increasing 
the number of observations within data sets, leads to increasingly generalizability among 
evaluation findings. 

Furthermore, the generalizability of evaluation findings is greatly enhanced by the 
number of clients who participate in the innovative treatment and the standardization of 
treatment and evaluation protocols used across treatment sites. Within this context, the most 
pressing need for standardization is the specification and collection of similar information about 
treatment services and clients. Similar information about different treatment approaches among 
different populations rnakes it possible to address the broad knowledge and national policy 
questions about treatment effectiveness, treatment efficiency and treatment replicability (Anglin 
& Hser, 1992). 

3. NEED FOR SYSTEMATIC COLLECTION OF TREATMENT, COST AND 

CLIENT INFORMATION 

Prior evaluation efforts have collected extensive information about either the 
implementation of the intervention or about the clients who received intervention services. Few 
evaluations collected both intervention and client data systematically which then could be linked 
together during analysis, and even fewer collected cost and performance data systematically 
which could be incorporated within the overall analysis. Also, the systematic collection of client 
follow-up data, which is essential to determining treatment outcomes, has received even less 
emphasis (Pietrzak et al., 1990). 
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The systematic collection of treatment, cost and client data requires: (1) efficient use of 
resources: collecting data on one segment of treatment provides minimal additional insight into 
the entire treatment experience; therefore collecting comprehensive treatment, cost and client 
data, including follow-up data for a requisite sample of clients, is a more efficient use of 
resources; and (2) sufficient technologies and measures: standardized, reliable measures for 
service delivery imit variables, clinical components of treatment, treatment costs and client 
outcomes are essential (Longbaugh et al., 1993). 

The CSAT evaluation efforts have confronted many of the coordination, comparability, 
methodological and technical issues. As a result of these experiences, CSAT evaluations have 
included developmental components and produced or enhanced the following: 

■ An approach to evaluation that facilitates continuous knowledge development and 
treatment services improvement 

■ Conceptual designs for coordination among treatment and evaluation activities, cost 
analysis, and performance measurement 

■ Measurements for service delivery imits and clinical components and quantifying and 
linking these measures to client outcomes 

■ Standardized approaches to specifying information needs including evaluation 
measures, variables, definitions and corresponding instrumentation 

■ Automated data collection and management tools 

■ Models for systems, services, clinician and client data analyses and reporting 

■ Follow-up data collection activities. 

The integration of these concepts, measures, methods and tools is the subject of this document. 
The next section provides the evaluation framework for the integrated evaluation methods 
approach. 

4. NEED FOR OVERARCHING APPROACH TO DATA INTEGRATION 

CSAT evaluation activities between 1990 and 1998 resulted in invaluable lessons learned 
about substance abuse treatment services, about evaluation methodologies, and about 
implementation of sound evaluation methods for CSAT treatment demonstrations. A resounding 
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lesson learned from the eight years of CSAT experience is that there is a serious gap in the 
comparability of substance abuse treatment services, clinician, cost, and client outcome data, 
among local study sites, and across CSAT program activities. This lesson leads to the conclusion 
that CSAT needs to provide an overarching approach that facilitates the integration of data across 
evaluation activities and among CSAT programs. The need for data comparability and 
integration has been clearly identified by CSAT-sponsored evaluators and is well-grounded in 
the professional literature, as the earlier literature citations suggest. This need also has been 
identified by the CSAT-convened panels of nationally renowned substance abuse treatment 
researchers and evaluators. 
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The lessons learned from prior CSAT evaluation activities are directly relevant to current 
and future knowledge-generating activities. All such efforts will require coordination among 
sponsors and study sites; integration of evaluation and clinical practices; interrelational 
evaluation approaches and methods; comprehensive data collection and analyses across 
programmatic efforts; and products that appeal to a broad audience and range of consumers 
(policy makers, treatment managers, clinicians, evaluators, and the general public). 



The conceptual and methodological products from prior CSAT evaluations provide an 
analytic framework which makes sense of the complexities inherent in substance abuse treatment 
evaluation and provides an analysis strategy for moving forward. The CSAT evaluation 
framework is presented in Exhibit IV- 1 and includes: 



■ Evaluation structures and processes which address treatment/evaluation coordination 
issues and the need to integrate substance abuse treatment funding, delivery, 
monitoring, evaluation data collection and analysis, and production of evaluation 
products 

■ Evaluation approaches which are interrelated and comprehensive to fully address the 
complexities of substance abuse treatment and the need for measures of substance 
abuse treatment efficacy and efficiency 

■ Evaluation strategies which take into account all of the realities of clinical services 
and client treatment experiences; strategies are needed to assist researchers and 
evaluators to disentangle the components within the “black box” 

■ Specification of evaluation data necessary to adequately support the full inventory of 
evaluation measures and the full range of analytic techniques. 

Each of these analytic framework components is further described below. 
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CSAT Evaluation Analytic Framework 
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Substance Abuse Treatment Evaluation Framework 



1. EVALUATION STRUCTURE AND PROCESS 

The importance and value of integrating ongoing evaluation activities within a system for 
treating substance abuse problems is widely recognized by CSAT, evaluators and providers alike. 
Also widely recognized, however, is that current knowledge and state-of-the-art approaches are 
often underutilized for adjusting and improving treatment systems as well as provider treatment 
services (Guba et al., 1989). Specifically, it is recognized that: 

■ The quality and quantity of evaluation information needs improvement 

■ Training and technical assistance on evaluation are needed to support service 
providers 

■ A shared sense of evaluation direction and priorities are needed among Federal and 
state staff, evaluators and service providers 

■ Federal agencies can provide leadership in the substance abuse treatment field by 
promulgating state-of-the-art evaluation concepts, approaches and tools. 

To begin addressing these needs, CSAT developed a self-adjusting treatment evaluation 
model which provides a framework for integrating evaluation within treatment planning, 
management, operations and service delivery. Through cycles of feedback and adjustment, this 
multi-disciplinary approach continuously incorporates knowledge gained from evaluation 
activities within the substance abuse treatment system or at a specific treatment site so as to 
improve client services (Devine et al., 1 999). 

A related CSAT developmental effort resulted in the capacity building model for teams 
made up of Federal sponsors, evaluators, treatment providers and other staff involved with the 
provision and/or evaluation of services. This CSAT tailored approach to team building was 
designed to assist Federal sponsors and treatment providers in building organizational structures, 
skills, behaviors and attitudes so as to successfully integrate the self-adjusting evaluation lessons 
within day-to-day management and service provision activities. To successfully use the model, 
all team members must work together and incorporate an understanding of sound evaluation 
principles; participate in planning and implementing evaluation components; collect and learn 
from new information, and implement and manage change in response to the new information 
(Moore et al. 1999). 
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Thirdly, CSAT presented a performance measurement model which provides tools and 
processes for measuring treatment outcomes and ensuring the ability to compare outcome 
measures across programs or against a standard that uses severity of client substance abuse as the 
analytic construct. This model will also support the use of performance measurement data by 
individual treatment providers to assess and monitor their own performance against pooled 
performance measurement data (Harwood et al., 1998). 

Taken together, these conceptual models and corresponding tools provide a firm 
foundation for establishing an evaluation structure and process among the Federal sponsors, 
evaluators and substance abuse treatment providers. These conceptual tools significantly address 
the issues of coordinating treatment and evaluation planning, implementation, and operations and 
the issues of integrating clinical, performance and evaluation information. 

2. EVALUATION APPROACHES 



The professional and academic fields of evaluation research have been evolving over the 
past two decades and are increasingly operationalizing the associated scientific and applied 
concepts. Major evaluation concepts include: 



■ Process evaluation — evaluation activities related to an assessment of a treatment 
provider’s operations; increasingly becoming synonymous with an assessment of the 
degree of conformity to the design (also termed: implementation evaluation). 

■ Impact or outcomes evaluation — evaluation of whether and to what extent a 
treatment approach or bundle of services causes changes in the desired direction 
among the target population. 

■ Cost analysis — the identification and analysis of all resources needed for a treatment 
provider’s operations; studies of the relationship between treatment costs and 
treatment outcomes, with both costs and outcomes expressed in monetary terms 
(Rossi et al., 1993). 

Evaluation professionals recognize the need for including all of these approaches when designing 
and implementing evaluation studies; public recognition, particularly with respect to funding 
complex evaluations, lags somewhat behind (Patton, 1987). The accumulation of CSAT 
evaluation experiences however fully supports the need to include the full range of evaluation 
approaches and the need to fully coordinate and integrate these approaches. The most recently 
stated CSAT goals further emphasize this need; CSAT aims to identify exemplary treatment 
practices that are cost efficient approaches to substance abuse treatment for populations with 
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problems of dependency and to assess the replicability of these treatment approaches (Steckler et 
al„ 1992). 

All of the national, and multi-site/cross-site evaluations directed by CSAT’s Program 
Evaluation Branch included process, impact, and cost analyses, and whenever possible, cost 
efficiency and cost benefit analyses. CSAT learned, however, that inclusion of these approaches 
is complex. Complexities emanate from the lack of standard definitions for substance abuse 
treatment services, expected client outcomes and cost components. Also, CSAT learned that the 
technical expertise needed to accommodate these evaluation approaches rarely rests with one 
professional evaluator who has both the breadth of treatment services and evaluation experience 
and the in-depth qualitative, statistical, economic, and cost analysis skills. Hence, CSAT uses 
evaluation teams to support its evaluation activities. 

To address the challenges of incorporating process, outcomes and cost analyses within 
the national evaluation efforts, CSAT sponsored the development of the concept paper: Adding 
“Value ” to CSAT Demonstrations: The What, How and Why of Cost Analysis and further tested 
the application of The Uniform System of Accounts and Cost Reporting for Substance Abuse 
Providers. The concept paper recognizes that rapid and dramatic changes are being made to the 
management, delivery and financing of substance abuse treatment services and that these changes 
place significant demands on publicly funded treatment systems as well as the services provided 
at treatment sites. To meet these challenges, meaningful evaluations must incorporate cost 
analysis along with process and outcome evaluations. The concept paper provides the rationale 
for this position and demonstrates the adoption and application of the Uniform System of 
Accounts and Cost Reporting for Substance Abuse Providers (Capital Consulting Corporation, 
1993, 1998; Harwood et al., 1998). 

Within this area, CSAT also developed a standard approach to developing evaluation 
plans which include process, outcomes and cost analysis and a standardized assessment protocol 
for reviewing evaluation proposals developed under the CSAT auspices. These models for 
evaluation plaiming were instrumental in furthering CSAT’s goal of providing essential 
standardization across evaluation activities at local study implementation sites. 

3. EVALUATION STRATEGIES TO FULLY CAPTURE TREATMENT 

EXPERIENCES 

CSAT has always recognized that substance abuse treatment is an evolving field of 
clinical, rehabilitative and social interventions and that the evaluation of substance abuse 

O SATACTRT END\IEM\OVERVIEW\IEM\DATA\IEMDOC.WPD 

ERIC 



31 



NEDTAC, Page 22 



Substance Abuse Treatment Evaluation Framework 



treatment requires an in-depth analysis of the range of treatment services both within and across 
treatment providers. CSAT also has recognized that the evaluation of substance abuse treatment 
services is confronted by multiple complexities, including: 

■ Substance abuse/dependency is associated with interrelated physical, biological, 
psychological, genetic, societal causes and variant medical diagnoses 

■ Substance abuse treatment encompasses multiple, overlapping and interrelated 
interventions related to treatment philosophies, modalities, therapies, models, and 
delivery mechanisms and staff 

■ Substance abuse patterns and treatment service patterns vary by geographic location, 
population characteristics, and socio-economic conditions (Wellisch et al., 1991). 

From the beginning, CSAT determined that service provider analysis must be conducted at the 
level of the service delivery unit — a single treatment modality provided at a single site (NORC, 
1997). As further evidence of client treatment experiences emerged, CSAT recognized the 
following concepts and their implication for substance abuse treatment evaluation: 



■ Treatment episodes — typically clients enter and exit treatment programs multiple 
times; this recognition reshaped CSAT’s concept of treatment discharge as an end 
point for treatment experiences and as a starting point for follow-up data collection 
and the measurement of treatment outcomes and impacts. 

■ Treatment “bundles” — complementary to the SDU measurement concept is the fact 
that treatment episodes frequently involve a continuum of care from detoxification 
to aftercare and that treatment clients may receive a “bundle” of services from several 
SDUs during one treatment episode. 

A fuller description of these treatment experience concepts and the implication for substance 
abuse treatment evaluation is provided below. 

3.1 Defining the Service Delivery Unit 

CSAT is interested in determining which types and combinations of treatment yield the 
best outcomes for specific populations and/or individuals with differing substance dependencies, 
demographic characteristics and/or treatment histories and experiences. Therefore, CSAT is 
guiding evaluations away from the view of treatment as a “black box” toward the scientific 
identification and measurement of treatment service components, treatment dosage, and 
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treatment experiences (Anglin & Hser, 1992; NORC, 1997; Saxe & Shusterman, 1991). To this 
end, CSAT is working to standardize the identification of and subsequent data collection about 
treatment modality, philosophy, setting, services provided, treatment organization, and clinical 
and staff characteristics and the interrelational impacts of these variables on client behaviors, 
attitudes and outcomes. 

This approach has resulted in CSAT’s creation of the Service Delivery Unit (SDU) 
concept and the operationalization of the concept with data specifications and definitions 
(NORC, 1997). An SDU is defined as a single site offering a single treatment modality. The 
SDU is one of the independent variables in an analysis of the impact of substance abuse 
treatment on treatment clients. While the client is the unit of analysis, the SDU is the unit of 
comparison. 

Inclusion of the SDU approach within substance abuse treatment evaluation is designed 
to continue the refinement of evaluations that attempt to answer the question: Which types of 
substance abuse treatment are most effective for a given population? This approach must be 
supplemented by client-specific treatment information. Substance abuse treatment providers 
may or may not provide all of the services associated with each SDU, to the same degree, to all 
of the clients who enter these SDUs. Collection of treatment experiences and “dosages” per 
client therefore is the most accurate and reliable approach to addressing the treatment 
effectiveness issue. 

3.2 Defining Treatment Episodes 

CSAT has always recognized that substance abuse treatment and recovery is a process of 
inter-related diagnostic, treatment, and assessment/reassessment experiences. In the mid-I990’s, 
CSAT adopted a substance abuse treatment model which provides a conceptual framework for 
understanding substance abuse treatment experiences and processes. This model, presented in 
Exhibit IV-2, graphically depicts the following treatment planning and selection phases: 

■ Phase I: Identification — involves determining an individual’s substance abuse 
problem and the severity (mild, moderate, severe) 

■ Phase II: Triage Referral — involves active referral to the appropriate treatment 
approach, where appropriateness is determined through mutual exploration with 
clients of their perceptions, needs, and desires for the type of treatment and setting 
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Exhibit IV-2 

Matching Treatment Settings, Strategies, Goals, and Modalities' 




'Adapted from: W.R. Miller (Matching Individuals With Interventions) and the model developed by the lOM Committee for the Study of Treatment 
and Rehabilitation of Alcoholism and Alcohol Abuse. 
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■ Phase III: Treatment Entry — involves the treatment provider’s response to clients’ 
immediate needs 

■ Phase IV: Initial Treatment Processes — involves implementing specific procedures to 
prevent treatment drop-out and to provide preparatory treatment experiences to 
relieve symptoms and to ensure motivation to continue treatment 

■ Phase V: Goals and Methods Selection — involves working with the client to 
collaboratively develop long range treatment plans 

■ Phase VI: Treatment Maintenance and Monitoring — involves regular review of 
progress toward goals, determining whether specific treatment methods are being 
adequately carried out, and redefining the goals and methods when necessary 

■ Phase VII: Termination and Follow-up — involves assessment of gains achieved and 
maintained, with an end to formal treatment occurring when the goals are reached 
(Diesenhaus, 1995). 

Building on the above treatment model, CSAT further recognizes that substance abuse 
treatment and recovery may include multiple treatment experiences. Typically, individuals with 
substance dependency seek out and receive a series of treatment services over time. Empirical 
evidence which supports this understanding underscores CSAT’s perspective that substance 
abuse treatment evaluation recognize each treatment experience as a treatment episode and that 
clients may participate in more than one treatment episode. From a evaluation design 
perspective, evaluators must be able to isolate treatment episodes per client when treatment is 
episodic and take into account these episodes when defining treatment exit and conducting 
follow-up data collection. Evaluators also must determine and then assess the combined effect of 
combining treatment episodes over time (Hubbard et al., 1989). 

3.3 Defining Treatment Intensity 

Substance abuse treatment providers, together with evaluators and researchers, must also 
identify the treatment intensity, as an important dimension of treatment dosage. A conceptual 
model for assessing the intensity of need and corresponding intensity of service is presented in 
Exhibit IV-3 on the following page. The model identifies five levels of need for substance abuse 
treatment, including: none, mild, moderate, substantial, and severe. For individuals who have 
no need for alcohol and other drug treatment, the appropriate service would be prevention. 
Individuals with mild to moderate substance abuse problems are candidates for brief 
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Exhibit IV-3 

Substance Abuse Need and Treatment Service Intensity 




interventions. Individuals with substantial to severe problems are candidates for specialized 
substance abuse treatment (Diesenhaus, 1995). 

3.4 Defining Treatment “Bundles” 

The detailed measurement of a client’s substance abuse treatment experiences also has 
relevance to the individual’s treatment episodes. Substance abuse treatment has evolved and 
currently reflects the step-down (and step-up) model of substance abuse treatment, depending on 
client need. When this process is integrated in accordance with a treatment plan and there is no 
break in the treatment experiences, the approach is termed “continuum of care” and viewed as 
one treatment episode by clients, providers, treatment funders and evaluators. When clients 
move in and out of treatment and when there are elapse times between treatment experiences, the 
treatment is termed “episodic care” and each treatment experience is considered one treatment 
episode (Breslin, 1 997). These concepts are graphically depicted in Exhibit IV-4. 

CSAT evaluators must be able to identify “treatment bundles” which result from case 
managed stepping up or stepping down of treatment intensity. In practical terms, the recognition 
of episodic care and the recognition of treatment bundles necessitates the collection of SDU- 
specific treatment data by date of treatment entry and treatment exit. The collection of these data 
will support a continuum of care analysis at the client level (Landry, 1995). 



er|c 



J:\CSAT\CTRT_END\IEM\OVERVIEW\IEM\DATA\IEMDOC.WPD 



36 



NEDTAC, Page 27 



EXHIBIT IV-4 

Schematic Presentations of Treatment Episodes, 
Continuum of Care and episodic Treatment 
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Substance Abuse Treatment Evaluation Framework 



4. SPECIFICATION OF DATA REQUIREMENTS 

Thus far, the CSAT evaluation analytic model recognizes the need for and provides 
concepts, methods and tools to: 

■ Ensure fully coordinated substance abuse treatment efforts which integrate clinical 
and evaluation information to ensure continuous knowledge generation and treatment 
service improvement 

■ Support the full range of evaluation approaches including process, outcomes, cost 
analyses, and performance indexing to ensure comprehensive evaluation 

■ Disentangle and fully specify treatment experiences including continuum of care, 
treatment bundles and episodic care to ensure that treatment services evaluation 
examines the relationship between client outcomes and services received. 

The evaluation framework also recognizes a fourth issue; namely, the need for comprehensive 
evaluation data to accommodate all treatment analysis requirements. A complete evaluation 
requires comprehensive treatment services, client and cost data; therefore, data issues underline 
all of the evaluation components, listed above (Moras, 1993). The contribution of CSAT’s 
evaluation experiences to the analytic framework here is the provision of a data specification and 
data collection framework to guide future efforts. The CSAT data framework includes: (1) time 
frame for data collection activities, and (2) types of data necessary for comprehensive 
evaluations. 

4.1 Types of Evaluation Measures and Data 

CSAT national substance abuse treatment evaluation studies consistently required data 
collection for the various dimensions associated with substance abuse treatment including 
systems, service delivery unit, costs and clients/comparison groups. These dimensions are 
described below. 

Systems level measures are essential if knowledge about the treatment field is to be 
meaningful. Systems measures capture information on the professional or community “system” 
within which the substance abuse treatment operates. The Criminal Justice Treatment Network 
program, for example, operates within the justice system, the treatment system and the broader 
community of public service systems. Similarly, the HIV Outreach program operates within the 
community “linkage” system — ^those agencies, particularly substance abuse treatment providers. 
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with whom the outreach program must establish working relationships to meet overall outreach 
and treatment access goals (Chen, 1 990). 

There is a growing literature on program evaluation that is based on the systems model 
(see Appendix D); this model presumes an integration of systems that directly or indirectly 
support substance abuse treatment. The model focuses the evaluation efforts on system-wide 
growth and survival including resource acquisition, environmental adaptation, and internal and 
external conflict resolution (Chen, 1990). Systems level measures may be inappropriate for 
evaluations which focus solely on individual treatment providers or treatment approaches for 
specific populations. Given the complexity of systems level evaluation, the current effort does 
not attempt to address this type of evaluation. 

Service Delivery Unit measures are critical to evaluation of treatment services and 
include information about the provider’s substance abuse treatment structure. The treatment 
structure shapes the clients’ substance abuse treatment experiences within a specific episode of 
care. SDU measures include treatment services design, philosophy, setting, implementation 
experiences, target population, clients served, components of treatment services, assessments, 
case management, treatment approaches, referrals and continuing care (NORC, 1997). 

Cost measures are essential to determining the efficiency of substance abuse treatment. 
One cornerstone of CSAT’s evaluation strategy is that cost analysis receive much greater 
emphasis in the evaluation of substance abuse treatment operations and outcomes. A cost 
analysis component allows the substance abuse treatment field to more fully capitalize on the 
knowledge being generated about substance abuse treatment services and apply it to treatment 
operations. CSAT recognizes that the value of individual substance abuse treatment cost 
analyses would be greatly enhanced by the ability to pool locally collected cost data and develop 
a national perspective on the relationship between the cost of services and service outcomes. 
Therefore, CSAT advocates a common list of cost and financial data to be collected locally in 
support of national analyses (Harwood et al., 1998). 

Clinical staff measures are critical to an understanding of treatment effectiveness since 
the clinical staff serve as the connection between the treatment services structure and the client’s 
treatment services experience (Lettieri, 1992). 

Client level measures are critical to an understanding of what types of treatment are best 
suited for a specific type of client. These measures include data on demographics and historical 
behavior; data on services actually received and length of stay, and data to support treatment 

O J:\CSAT\CTRT_END\IEM\OVERVIEW\IEM\DATA\IEMDOC.WPD 

ERIC 



40 



NEDTAC, Page 30 



Substance Abuse Treatment Evaluation Framework 



outcome measures. These measures fall within eight domains including demographic 
characteristics, substance use, juvenile/criminal justice involvement, mental health, physical 
health, education, employment, and social functioning. Client measures also include level of 
functioning and all treatment experiences including actual “dosages,” treatment exit, reasons for 
treatment termination, and experiences with other social and support services (Babor & Frawley, 
undated). 

Control or Comparison group(s) are essential to provide evidence that the treatment 
services have had an impact on the client behaviors. Control/comparison group(s) are designed 
to match the client group on all dimensions. The client level measures are used for the 
control/comparison group(s) and include all of the variables described above (Orwin, 1998). 

Building on the prior evaluation experiences, CSAT developed data lists which are 
included in the Minimum Evaluation Data Set (MEDS) for the service delivery unit, clinical staff, 
cost, and clients/comparison groups. The primary purpose of the MEDS is to assist CSAT- 
fiinded evaluations, in particular national evaluation studies, and knowledge-generating activities 
in planning and implementing comprehensive evaluations. 

4.2 Data Collection Time Points 

CSAT is interested in all aspects of treatment including treatment need, access, entry, 
participation and exit circumstances, and treatment outcome and impact. CSAT is committed to 
collecting client data for each of these treatment-related events: treatment referral; treatment 
services and dosage per client; treatment exit; continuing care; and follow-up. 

Further, CSAT is interested in gathering information about the clients who do not 
complete the treatment “flow.” Namely, clients who are referred but who do not seek treatment 
or are not accepted into treatment; clients who are accepted but do not complete treatment; and 
so on. Obtaining data on client flow at each juncture is critical to gaining insight into the need 
for and delivery of substance abuse treatment services. Exhibit IV-5 presents a diagram to 
demonstrate the “universe” of client data needed for each treatment episode. 
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Exhibit IV-5 

Universe of Client Data Per Treatment Experience 




Individuals who abuse substances and who are referred to substance abuse treatment and 
become treatment clients engage in a process that is graphically illustrated in Exhibit IV-6. As 
displayed, clients typically come to treatment via a referral or a contact from within their 
community. To determine eligibility and need for treatment, potential clients participate in an 
intensive intake assessment. Depending on service availability, clients then enter substance 
abuse treatment where they receive services. When treatment is completed or when clients 
otherwise leave treatment, treatment exit marks the end of a treatment episode. Many 
substance abuse treatment programs also include continuing care which usually is designed to 
facilitate re-entry to the broader community. Follow-up data collection then occurs so as to 
determine the client outcomes and impact of the treatment experience(s). 

In summary, this document recognizes that all data necessary to describe an episode of 
care are critical. Therefore, data have been specified at five analytic points: initial contact, 
treatment intake, within treatment, treatment exit, and follow-up. 
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V. Integrated Evaluation Methods (IEM) 

The CSAT evaluation analytic framework was constructed on the basis of accumulated 
experiences in conducting national and multi-site/cross-site evaluations by nationally and 
internationally known treatment services evaluation professionals. These experiences 
demonstrate that fully evaluating substance abuse treatment services to determine the 
implementation processes, client treatment experiences, client outcomes, and treatment costs is 
essential, yet challenging. 

To successfully evaluate national programs which have been implemented among several 
program locations requires: (1) comprehensive evaluation methods; (2) comprehensive data 
collection from multiple sources to support the evaluation analyses; (3) careful coordination 
among the funders, treatment service providers and evaluators; and (4) thoughtful integration of 
evaluation findings within the clinical program (Monahan et al., 1996). As prior CSAT 
evaluators confronted these challenges, CSAT directed the development and application of 
analytic concepts, evaluation methods, and data collection tools to address each challenge. The 
prior CSAT evaluations not only provided the impetus for these developmental efforts but also 
provided a testing ground for new approaches and processes. Since each concept and tool was 
designed to resolve an evaluation challenge, each concept and tool was applied and then refined 
based on the application experiences. This paper and the IEM reflect and incorporate these 
experiences, gained over the last decade; this section describes the IEM. 

The purpose of this section is to describe the IEM approach which integrates the 
previously developed evaluation concepts, methods, and tools within an evaluation structure 
which is informed by both the analytic framework described above and a generic treatment 
services evaluation process. This section begins with a description of the evaluation process, its 
interrelationship with the CSAT evaluation analytic framework, and its applicability to the 
national treatment services evaluations and to the knowledge-generating process. A description 
of the evaluation concepts, methods and tools is then presented in accordance with the integrated 
evaluation methods structure. 
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1. DEFINING THE EVALUATION PROCESS 



As indicated earlier in this paper, treatment services evaluation is a specific type of 
applied social science research, and, as with any research, its execution involves following a 
standard set of tasks and activities (Pietrzak et al., 1990). These tasks are included in all types of 
evaluation and they generally occur in the following order: 



■ Planning the evaluation — the planning activities include setting the evaluation 
goals and objectives that determine the overall parameters of the evaluation; 
typically, the evaluation plan then lays out how the evaluation will assess whether the 
expected results were achieved; the evaluation plan incorporates the sponsor’s 
philosophies and policies. 

■ Specifying the evaluation questions and measures — the evaluation questions are 
the apex of the evaluation and provide specific criteria which guide the evaluation and 
prescribe the evaluation measures, variables, data, data collection and data analysis. 

■ Selecting the evaluation design — the evaluation design sets forth the overall strategy 
for examining the evaluation questions, measurement, controls, validity and 
reliability, sample needed, design effects, and generalizability of findings. Treatment 
service design and evaluation conditions determine appropriate evaluation design 
selection. 

■ Developing the study methodology and evaluation implementation — this phase of 
the evaluation guides the selection of data collection methods and includes data 
collection procedures, data management systems, client confidentiality forms and data 
collection guides. 

■ Developing the data requirements and instrumentation — the data collection 
instruments are measuring devices used to collect information in a standardized, 
comparable format. The data requirements flow from the evaluation questions and 
measures. During this phase of the evaluation, the data requirements are organized by 
types of data including: systems level data, treatment service level and SDU data, 
clinical data, cost data, and client data. Data collection instruments are then 
developed or selected from a standard inventory of instrumentation. Also, during this 
phase, the client confidentiality procedures, data management systems, and data 
collection procedures are developed. 

■ Analysis plan — the analysis plan is dependent on the evaluation design, evaluation 
questions, the evaluation data collected, and the products the evaluation will produce. 
For CSAT evaluations, both process and outcome analyses are expected. 
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■ Collecting the data — data collection will be initiated and completed during this 
phase of the evaluation; the data collection phase includes the development of data 
management processes and tools including quality control procedures. 

■ Analyzing the data — data analyses usually involve some if not all of the multiple 
levels of comparison and are governed by the analysis plan and intended products and 
target audience(s). 

■ Reporting the evaluation flndings — reporting the evaluation results is equal in 
importance to the other evaluation processes since it is through the presentation and 
publication of evaluation findings that the knowledge gained from the evaluation can 
be disseminated and then applied within the full body of substance abuse treatment 
knowledge. 

Most if not all of the components identified within this generic evaluation process are included in 
past, present and future CSAT national evaluation and knowledge-generating efforts. The 
purpose in laying out the evaluation process is to provide an evaluation model for incorporating 
the CSAT evaluation analytic concepts and the CSAT developed conceptual and methodological 
applications, as described below. 

2. BUILDING THE INTEGRATED EVALUATION METHODS APPROACH 

The CSAT evaluation framework provides a strong foundation for an integrated 
evaluation methods package to support CSAT knowledge-generating and application efforts. 

The interrelationships of this framework is provided in Exhibit V-1 . As shown, the four 
components of the evaluation framework provide the foundation for tailoring the evaluation 
activities to the CSAT environment. The CSAT requirements for coordination, comprehensive 
evaluation approaches, evaluation strategies for the treatment services, and specified data 
requirements to support analyses have implications for each of the evaluation activities within 
the overall evaluation process. 

A CSAT tailored evaluation plan and the specification of evaluation questions and 
measures forms the core of the steps associated with the Guidance for Application (GFA) for 
future knowledge-generating grant programs and Statements of Work (SOWs) for national 
evaluation studies. Grant applications and contract proposals will yield stronger and more 
productive evaluation implementation by using the concepts and tools associated with integrated 
evaluation designs, study methods, and data requirement specification. Finally, as CSAT project 
officers monitor the grant, cooperative agreement, and contract implementation, their activities 
will be enhanced by the CSAT-developed guidance on data collection, analysis, and reporting. 
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Exhibit V-l 

Integrated Knowledge Generating Process 
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3. SPECIFYING THE INTEGRATED EVALUATION METHODS COMPONENTS 



This section describes the specific components of the Integrated Evaluation Methods 
package. Each of the components of the package, including examples of the evaluation products 
developed by CSAT to assist in the implementation of the integrated approach, are described 
below. 

3.1 Concepts to Support Evaluation Planning 



CSAT has articulated an integrated approach to treatment evaluation and integrative 
methodologies, including building an evaluation team, applying state-of-the-art cost analysis and 
performance measurement and providing continuous feedback among program planners, 
providers and evaluators. Complete descriptions of these concepts and their integrative 
relationships are provided in the documents listed below: 



■ Integrated Evaluation Methods (lEM) integrates the concepts, processes, procedures, 
and tools developed in the concept papers listed below. 

■ The Self-Adjusting Treatment Evaluation Model (SATEM) provides a framework for 
evaluation activities to be integrated within planning, management, operation and 
service delivery activities. 

■ Building Team Capability to Fully Implement and Utilize the Self-Adjusting 
Treatment Model provides overall guidance to assist service providers in building an 
organizational capacity for integrating the SATEM. 

■ Adding Value to CSAT Demonstrations: The What, How and Why of Cost Analysis 
introduces tools and processes that enable the evaluation team to measure costs and to 
demonstrate the value of their services. 

■ Performance Measurement for Substance Abuse Treatment presents tools and 
processes for measuring treatment outcomes and ensuring the ability to compare 
outcome measures against a standard. 

■ Client Levels of Functioning as a Component of Substance Abuse Treatment Services 
Evaluation is a description of the rationale and methods for assessing client level of 
functioning and recommended core EOF data elements that could help to measure the 
effectiveness of treatment services received. 
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CSAT provides additional tools to support evaluation planning: the Substance Abuse Treatment 
Evaluation Policy Notebook and the Substance Abuse Treatment Evaluation Resource Notebook. 
The policy notebook describes Federal guidelines and SAMHSA policies that impact Federally 
sponsored evaluations. The resource notebook is a rich source of information to assist in 
evaluation plarming activities. 

3.2 Guidance for Selecting the Evaluation Design 

Process evaluation describes substance abuse treatment providers, services, and 
procedures. CSAT advocates the integration of process and outcome evaluation approaches. A 
detailed process evaluation supports more in-depth analysis of (1) the complex interrelationships 
among the often enigmatic variables that affect treatment variables; (2) the association between 
service delivery components; and (3) anomalies in the treatment outcomes revealed by the 
outcome evaluation. In essence, outcomes carmot be fully explained without an understanding of 
the processes that yielded the outcomes (Judd, 1987). To support the development of a process 
evaluation design, CSAT developed a Guide to Process Evaluation for Substance Abuse 
Treatment. 

Building on evaluation technologies developed previously, the CSAT logic model tool. 
Using Logic Models in Substance Abuse Treatment Evaluations, provides a description of the 
techniques, the variations of logic modeling in response to knowledge-generating GFA and SOW 
specifications, a review of the logic model literature, and several “how to” examples. The logic 
model document also describes a CSAT -developed tool entitled Data Maps. Data mapping 
aligns the program goals and objectives with the evaluation goals and objectives so as to specify 
the evaluation questions and the measures necessary to answer the questions and fulfill the 
evaluation goals and objectives. 

The myriad of evaluation designs, which are grounded in the process of scientific inquiry, 
are only useful when applied appropriately. The availability of data, control and/or comparison 
groups, the timing of the evaluation, and the evaluation resources all influence the selection of 
the most appropriate evaluation design. To support this selection process, CSAT developed a 
technical assistance paper. Guide to Selecting an Outcome Evaluation Design for Substance 
Abuse Treatment Evaluations, which provides theoretical pinnings together with practical 
guidance for CSAT evaluations. 
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3.3 Developing Data Requirements 

Based on experience gained through past evaluation efforts, CSAT recognized the need 
for a uniform and integrated set of data variables. Entitled the Minimum Evaluation Data Set 
(MEDS), the data lists include variables on treatment service structure, services available, clinical 
components, the costs of providing the services and clients. The client variables include eight 
domains: demographic characteristics, substance use, juvenile/criminal justice involvement, 
mental health, physical health, education, employment, and social functioning. The MEDS also 
articulates the CSAT evaluation approach, including: 

■ Levels of data to be collected including: systems, SDU, clinical, unit cost, and client 
outcome data 

■ Data collection time frames 

■ Service Delivery Unit concepts and data 

■ Treatment service bundles analysis 

■ Cost and financial data collection analysis. 

Together, the MEDS components comprise the integration of essential CSAT evaluation 
philosophy, policies, concepts and procedures. 

CSAT has developed an additional tool to assist in the evaluation of substance abuse 
treatment services, the Substance Abuse Treatment Cost Analysis and Allocation Template 
(SATCAAT), a document that provides a user manual to collect and analyze treatment costs by 
unit of service for an SDU. This document provides a conceptual overview of the cost 
components that are being assessed and sample cost templates for use in an evaluation. 

3.4 Developing Data Collection Protocols and Instruments 

CSAT has developed data collection tools to parallel the concepts and variables of the 
MEDS. Those tools are collected in a document entitled Substance Abuse Treatment Services 
Evaluation Data Collection Instruments. These tools include: 

■ Protection of Human Subjects 

■ Service Delivery Unit (SDU) Description 
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■ Clinician Background and Practice Survey 

■ Cross-site Evaluation Protocol (Client), including Record Extraction Forms. 

Automated data management systems and tools have been developed for previous CSAT 
evaluation efforts; these systems and tools, which include data quality control mechanisms, are 
available for adaption to future efforts. 

3.5 Guidance for Follow-up Data Collection 

CSAT developed manuals and strategies to support high quality follow-up data collection 
among hard-to-reach populations, including: 

■ Staying In Touch: A Fieldwork Manual of Tracking Procedures for Locating 
Substance Abusers for Follow-up Studies 

■ Strategies for Follow-up Tracking of Juvenile, Homeless, and Criminal Justice 
System-Involved Substance Abusers: Overview and Bibliographies 

3.6 Analyzing the Data 

Given the CSAT emphasis on SDUs, clinical components, costs and performance 
measurement and client outcomes, it is essential that CSAT sponsored evaluations include 
comparable analytic concepts, constructs, models, assumptions and statistical techniques. To 
assist the adoption of comparable analytic approaches, the Guide to Substance Abuse Treatment 
Evaluation Data Analysis was developed. 

3.7 Reporting Evaluation Findings 

Given that a primary CSAT mission is the dissemination of substance abuse treatment 
knowledge, the reporting of evaluation findings must be appropriate to the audiences for these 
evaluation products. Therefore, CSAT developed a series of guides and templates for developing 
evaluation products for the following types of evaluation: process evaluation, outcome 
evaluation, cost analysis, performance measurement, replication studies, ethnographies, and case 
studies. These guides are collected in the document entitled Substance Abuse Treatment 
Evaluation Product Outlines Notebook The guides include: 
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■ Evaluation Plan Outline 

■ Interim Evaluation Report Outline 

■ Final Evaluation Report Outline 

■ Fact Sheet Template 

■ Evaluation Briefing Outline and Template 

■ Data Templates. 

4. INTEGRATED EVALUATION METHODS PACKAGE 

Each of these concept papers, methods papers, procedures and tools is presented in a 
separate document which, taken together, constitutes the Integrated Evaluation Methods (lEM) 
Package. The lEM and its relationship to the analytic framework and evaluation process is 
graphically presented in Exhibit V-2. 
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Integrated Evaluation Methods Package: 

A Guide For Substance Abuse Treatment 
Knowledge-Generating Activities — Executive Summary 

Since its inception, the Center for Substance Abuse Treatment (CSAT) has provided 
Federal leadership to improve substance abuse treatment accessibility, effectiveness, and 
efficiency. CSAT’s mission and activities have evolved from directly supporting treatment 
services to supporting knowledge-generating activities. This evolution is evident in the current 
Substance Abuse and Mental Health Services Administration policy on evaluation as described 
'm Evaluation Policy, SAMHSA, 1995. 

The need for an integrated model of evaluation and planning at SAMHSA is presented in 
“Evaluation in the Substance Abuse and Mental Health Services Administration,” Evaluation 
and the Health Professions, by Marsh, Jansen, Lewis, & Straw, 1996. CSAT also supports site- 
specific, cross-site, and national evaluations that have provided experience with a wide array of 
evaluation design and implementation methods. These experiences further supported the need 
for an integrated evaluation strategy and led to the development of a comprehensive set of 
evaluation products, including concept papers, technical assistance (TA) materials, and analytic 
tools. Collectively, these products are referred to as the Integrated Evaluation Methods (lEM) 
Package. The lEM Package organizes these products within an evaluation framework that is 
designed to support CSAT knowledge development and application goals. The evaluation 
framework itself was constructed on the basis of accumulated experiences among internationally 
known treatment service evaluation professionals. The lEM Package reflects and incorporates 
evaluation experiences gained over the past decade. 

Evaluation Framework and the Integrated Evaluation Methods Package 

National evaluation experiences have reinforced the fact that substance abuse treatment 
evaluation involves a standard set of tasks that generally occur in the following order: 

■ Planning the evaluation/knowledge-generating activities, which includes selecting 
the substance abuse treatment issue, identifying the theoretical foundation for the 
intervention, determining knowledge development program goals and implementation 
approach, and setting the evaluation goals and Objectives that determine the overall 
parameters of the evaluation 
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■ Selecting the evaluation design, which sets forth the overall strategy for establishing 
the process and outcome evaluation questions, measurement approach, and 
generalizability of findings 

■ Developing the data requirements, which flow from the evaluation questions and 
measures and include: SDU, clinician, cost, and client data 

■ Developing data collection instruments, which are based on the data requirements 
and are developed or selected from an integrated inventory of instrumentation 

■ Collecting the data, which includes developing data management processes and tools 
(including quality control procedures) and conducting the data collection activities 

■ Analyzing the data, which involves multiple levels of comparison and is governed 
by an analysis plan 

■ Reporting the evaluation findings, which includes evaluation knowledge 
dissemination and application within the field. 

The evaluation process outlined above provided a framework for the development of products 
related to these evaluation concepts and methods. Taken together, those products comprise the 
IBM Package. 

Integrated Evaluation Methods Products 

CSAT requested the development of a series of evaluation concept papers, TA materials, 
and tools to support and operationalize each phase in the evaluation of substance abuse treatment 
knowledge-generating activities. These items are included in the IBM Package. The concept 
papers are based on theoretical evaluation research constructs that have been adapted to 
substance abuse treatment services evaluation and knowledge-generating activities. The concept 
papers primarily support the evaluation planning phase and address such topics as the self- 
adjusting treatment evaluation model, cost analyses, and performance measurement. The TA 
materials and tools include specific evaluation methods that have direct applicability to substance 
abuse treatment knowledge-generating activities. The concept papers and TA materials that 
constitute the IBM Package are listed and briefly described in Bxhibit I. 
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1. Planning the 
evaluation/ 
knowledge- 
generating 
activities 


■ Integrated Evaluation Methods: A Guide for Substance Abuse Treatment 
Knowledge Generating Activities: Concept paper that describes the development of an 
evaluation framework, evaluation concepts, and TA materials to support the framework. 

■ Self-Adjusting Treatment Evaluation Model: Concept paper that describes an 
approach for integrating evaluation findings within treatment operations so as to adjust 
and improve service delivery. 

■ Building Team Capability to Fully Implement and Utilize the Self-Adjusting 
Treatment Evaluation Model: Concept paper to assist treatment providers in building 
capabilities to integrate the self-adjusting treatment model within day-to-day operations 
and service delivery. 

■ Adding ‘*Value” to CSAT Demonstrations: The What, How and Why of Cost 
Analysis: Concept paper on the need for and types of cost analyses for CSAT 
demonstrations and knowledge-generating activities. (The Lewin Group) 

■ Performance Measurement for Substance Abuse Treatment Services: Concept 
paper about the increasing importance of provider performance measurement and 
analyses and an explanation of the case-mix adjustment methodology. 

■ Client Levels of Functioning as a Component of Substance Abuse Treatment 
Services Evaluation: Description of the rationale and methods for assessing client level 
of functioning and recommended core LOF data elements that could help to measure the 
effectiveness of treatment services received. 

■ Substance Abuse Treatment Evaluation Policy Notebook: These materials are aimed 
at facilitating understanding of the SAMHSA policy for evaluation and federal 
regulations on client confidentiality and assisting evaluators to meet CSAT evaluation 
requirements. 

■ Substance Abuse Treatment Evaluation Resource Notebook: The notebook contains 
evaluation bibliographies and listings of organizations, hot lines, on-line data bases, and 
contact information for obtaining assistance in evaluating treatment services. 


2. Selecting the 
evaluation design 


■ A Guide to Process Evaluation for Substance Abuse Treatment Services: TA tool 
presenting purposes of process evaluation and the application of process evaluation 
methods to single site and multi-site treatment services. 

■ Using Logic Models in Substance Abuse Treatment Evaluations: TA tool describing 
logic model purposes and techniques for designing and planning the evaluation of 
treatment services. 

■ A Guide to Selecting an Outcome Evaluation Design for Substance Abuse 
Treatment Evaluations: TA tool describing overall strategies for developing 
evaluation questions, measurement, controls, validity/reliability, sampling, design 
effects, and generalizability of findings. (Battelle) 
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3. Developing data 
requirements 


■ Minimum Evaluation Data Set (MEDS): Core Data Lists: TA tool for developing a 
uniform set of variables and response categories for the service delivery unit (SDU), 
clinician, cost, and client evaluation measures. 

■ Substance Abuse Treatment Cost Allocation and Analysis Template (SATCAAT): 
User manual to analyze treatment costs by unit of service for an SDU. (Capital 
Consulting Corporation) 


4. Developing data 
collection 
instruments 


■ Substance Abuse Treatment Services Evaluation Data Collection Instruments: Data 
collection instruments that fully incorporate the MEDS and that have been field tested 
for validity and reliability, as follows: Service Delivery Unit (SDU) Description; 
Clinician Background and Practice Survey; protocols to collect Adult, Adolescent and 
Child (in treatment with parent) Client Data at Intake, During Treatment, at Treatment 
Discharge and Post Treatment; Adult and Adolescent Record Extraction forms; and a 
section on protection of human subjects and informed consent. 


5. Collecting the 
data 


■ Staying In Touch: A Fieldwork Manual of Tracking Procedures for Locating 
Substance Abusers for Follow-up Studies (UCLA): User manual to establish and 
implement client follow-up data collection systems and procedures. 

■ Strategies for Follow-up Tracking of Juvenile, Homeless, and Criminal Justice 
System-Involved Substance Abusers: Overview and Bibliographies, 1990-1998: 
Description of tracking techniques used to increase response rates for follow-up 
interviews with homeless and juvenile/criminal justice involved substance abusers. 


6. Analyzing the 
data 


■ A Guide to Substance Abuse Treatment Evaluation Data Analysis: Recommended 
methods and procedures for analyzing process, SDU, clinician, cost, and client 
evaluation data. 


7. Reporting the 
evaluation 
findings 


■ Substance Abuse Treatment Evaluation Product Outlines Notebook: Compendium 
of outlines for evaluation products including evaluation plans, interim evaluation reports, 
final evaluation reports, replication studies, case studies, and ethnographies. 
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CSAT Evaluation “Stakeholders” 

Evaluation “stakeholders” are individuals, groups, or organizations that have a significant 
interest in how well a program or activity functions. (See P.H. Rossi, H.E. Freeman, & M.W. 
Lipsey, Evaluation: A Systematic Approach, 6th Edition, 1999.) Within the context of the lEM 
Package, CSAT evaluation stakeholders include CSAT senior managers, CSAT project officers, 
and CSAT grantees and contractors including treatment service providers, coordinating centers, 
study sites, site-specific evaluators, and national evaluators. 

Utility of the lEM Package for CSAT Evaluation Stakeholders 

While the conceptual and TA materials were developed from the perspective of the site- 
specific and multi-site evaluator, the concepts and TA tools have important utility for CSAT 
managers, project officers, and treatment service providers. The stakeholder’s position 
determines the perspective and utility of the lEM Package concepts and tools. For example, a 
CSAT senior manager can use the lEM Package to acquire a comprehensive evaluation context 
for planning and funding the knowledge-generating activities, the project officer can use the lEM 
Package to ensure that GFA/RFP applications are complete and include a full complement of 
design, execution, and product components, and the site-specific and multi-site evaluators can 
use the lEM Package to ensure that evaluation designs, data collection plans, data analyses, and 
product development have a consistent evaluation framework and compatible data across 
program areas. The suggested utility of the lEM Package for CSAT evaluation stakeholders is 
summarized in Exhibit II. 
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Exhibit II 

Utility of IEM Package for CSAT Evaluation Stakeholders 



stakeholders/ 


CROli^NDRESPi^ 




SENIOR 

MANAGERS 


■ Policy development 

■ Issue identification for KD&As 

■ Grant/contract funding decisions 

■ Overall program management 

■ Sustainability 

■ Dissemination 

■ Long-term strategic planning 

■ Program designs 

■ KA activities 


■ Comprehensive evaluation framework 

■ Comprehensive evaluation components 

■ Roles and responsibilities for local/national 
evaluators as well as CSAT/grantee staffs 

■ Guidance for evaluation designs and 
products 

■ Standardized evaluation measures 

■ Logic models for program and evaluation 
design 


PROJECT 

OFFICERS 


■ GFA/SOW development 

■ Grant/contract application review 

■ Grant/contract monitoring 

■ Knowledge-generating products 

■ Identification and replication of 
promising practices 

■ Technical assistance assessment 


■ Guidelines for high-quality evaluation 
designs (process and outcome) 

■ Logic models for program and evaluation 
designs 

■ List of evaluation measures with 
instrumentation 

■ Guidelines for evaluation products 


GRANTEES: 
STUDY SITES 


■ Grant applications 

■ Project development, implementation 

■ Local evaluation management 

■ Local evaluation coordination 

■ Knowledge-generating product 
development 


■ Evaluation plan outline 

■ Process and outcomes evaluation designs 

■ SDU, clinician, cost, and client measures 

■ Roles and responsibilities for grantee 
provider/evaluator staff 

■ Guidelines for evaluation products 


GRANTEES: 

MULTI-SITE 

EVALUATORS 


■ Grant applications 

■ Comprehensive evaluation designs 

■ Evaluation implementation: 

- Data collection 

- Data analysis 

- Reporting evaluation findings 

■ Evaluation product development 


■ Evaluation concepts 

■ Logic models 

■ Evaluation designs 

■ Evaluation data requirements 

■ Data collection instrumentation 

■ Data collection process and procedures 

■ Data analysis 

■ Product development 


NATIONAL 

EVALUATORS/ 

SERVICES 

RESEARCHERS 


■ Contract applications 

■ Comprehensive evaluation designs 

■ Evaluation implementation: 

- Data collection 

- Data analysis 

- Reporting evaluation findings 

■ Evaluation product development 


■ Evaluation concepts 

■ Logic models 

■ Evaluation designs 

■ Evaluation data requirements 

■ Data collection instrumentation 

■ Data collection process and procedures 

■ Data analysis 

■ Product development 
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IEM products and other evaluation materials may be obtained from: 
http://neds.calib.com 
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Editor’s Note 



This document is one of a series of papers that describe CSAT’s approach to substance 
abuse treatment evaluation. The graphic below illustrates the continuous evaluation knowledge 
development and application process which characterizes CSAT’s approach. At the core is the 
self-adjusting treatment evaluation model which is the foundation. The model integrates 
continuous, state-of-the-art evaluation with planning, management, operation, and service 
delivery within a multi-disciplinary learning community. Implementation of this model requires 
building of team capabilities, appropriate, state-of-the-art performance evaluation and cost 
analysis, and assimilation of CSAT’s integrative approach to treatment evaluation and integrative 
methodologies. Each of these processes work together to ensure continuous improvement. 



Ensuring Continuous Evaluation Knowledge Development and 

Application 




Assimilate Approach to ^ w " Ensuring Continuous, State-of-the-Art Conduct 

Treatment Evaluation Evaluation Performance Evaluation 




O ;at\ctrt_end\iem\overview\iem\data\iem_app.wpd 



NEDTAC, Page B-1 



Appendix B 



Substance abuse treatment providers are increasingly called upon to demonstrate that they 
are delivering appropriate services, that those services have the desired impact, and that the 
services justify the costs. An ongoing process of evaluation and systems/services improvement 
integrated into the day-to-day operation of treatment providers is needed to do so. In addition, 
the evaluation and improvement process requires a multi-disciplinary team that includes 
treatment personnel, evaluators. Federal and State agencies, advocacy groups, funding agencies, 
and the community. Building team capability is integral to this approach. Treatment staff must 
be involved in knowledge development and application (i.e., planning and implementing 
evaluation efforts, incorporating changes in response to new knowledge, and sharing of 
findings). 
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Doug Anglin, Ph.D. 


Director, UCLA Drug Abuse Research Center, Neuropsychiatric Institute, UCLA; 
Director, California Drug Abuse Information & Monitoring Project; Adjunct Associate 
Professor of Medical Psychology, Neuro-psychiatric Institute, UCLA. 


Tom D’Aunno, Ph.D. 


Director of Mental Health Services Research Training Program and Associate Professor, 
School of Social Service Administration, Faculty Associate, Center for Health 
Administration Studies, University of Chicago; PI, “Managed Care in Drug Abuse 
Treatment,” “Drug Abuse Treatment System Survey,” NIDA; Instructor, “Social 
Intervention: Policies & Programs,” “Organizational Theory and Health Care 
Organizations.” “Health Policv”; Editorial Board Member. Health Administration Press. 
Journal of Health and Social Behavior. Medical Care Review and Research: Member. 
Academy of Management, American Sociological Association. 


Dan Griffith, Ph.D. 


Supervising Clinical Psychologist, Northwestern Memorial Hospital, Chicago; Associate, 
the Lewin Group; Co-principal investigator on six-year NIDA project ot assess 
developmental outcomes and service needs of children exposed prenatally to cocaine and 
other drugs; Evaluation Consultant on U.S. Department of Education grant to Florida 
A&M University to deyelop materials and resources for teachers educating children 
prenatally exposed to alcohol and other drugs; Clinical Psychologist, Northwestern 
University Medical School and National Association for Perinatal Addiction Research 
and Education. 


Patricia Harrison, Ph.D., LP 


Licensed Psychologist; Minnesota Dept, of Human Services; Project Director, State 
Performance and Substance Abuse Treatment Outcome Pilot Studies (CSAT); P.D., 
Statewide Prevention Assessment (CSAP); P.D. , Statewide Treatment Demand and 
Needs Assessment Studies (CSAT); P.D., Implementation of Uniform Alcohol & Drug 
Abuse Data Collection Systems; P.D. Minnesota Treatment Accountability Plan; P.Co-D., 
Minnesota Student Survey, Dept, of Education. 


Yin-Ing Hser, Ph.D. 


Associate Director, UCLA Drug Abuse Research Group of NPI, UCLA; Adjunct 
Professor, Dept, of Psychiatry & Biobehavioral Sciences, University of California, 
UCLA; Co-PI, DATOS; Drug Abuse Epidemiology and Prevention Research Review 
Committee, NIDA; Co-PI, NEDTAC/CSAT; PI and Committee Member, NIDA; IT 
consultant for Dept, of Health, Republic of China, Taiwan; Scientific Expert for 
Commission on Alcoholism & Narcotics, and the Dangerous Drug Commission Joint 
Planning & Budget Committee, L.A. County Drug & Alcohol Program Administration 
Office. 


Robert Hubbard, Ph.D. 


Director, North Carolina Office, National Development & Research Institutes, Inc. 
(NDRI-NC); Principal Investigator/Project Director, DATOS; Founding Member of 
Board of Directors for Governor’s Institute on Alcohol & Drug Abuse. 
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Michael Knox, Ph.D. 


Licensed Psychologist; tenured (Distinguished Service) Professor, Dept, of Community 
Mental Health, Louis de la Parte Florida Mental Health Institute, and Professor of 
Medicine, Dept, of Internal Medicine, College of Medicine, University of South Florida; 
Director, USF Center for HIV Education & Research; University of South Florida; 
Professor, Dept, of Community & Family Health, and Dept, of Psychology, University of 
Florida. 


Bill Luckey, Ph.D. 


Senior Research Psychologist and Program Director of Substance Abuse Treatment 
Research, RTI; PI, Managed Care, Treatment/Service Patterns and Outcomes, NIAAA; 
PD, Methadone Treatment Quality Assurance System, NIDA. 


Mary Ellen Marsden, Ph.D. 


Associate Research Professor, Institute for Health Policy, Brandeis University; Faculty 
Associate, NIAAA Training Grant on Alcohol Services Research; Adjunct Research 
Faculty, Brown University Center for Alcohol & Addiction Studies; Study Director, Five 
Alcohol & Other Drug Study Sites, Coordinating Center for Managed Care & Vulnerable 
Populations; Co-PI, Process & Outcome Evaluation of Correctional Substance Abuse 
Treatment; Co-PI, Massachusetts Substance Abuse Treatment Performance & Outcome 
Monitoring Pilot Study; Co-PI, Health & Performance of Military Women; Consultant, 
Revision of RWJ Chartbook on Substance Abuse. 


Jane Maxwell 


Director, Needs Assessment Department, Texas Commission on Alcohol and Drug 
Abuse. 


Bruce Rounsaville 


Director of Research, Division of Substance Abuse, Professor of Psychiatry, Department 
of Psychiatry, School of Medicine, Yale University. 


Connie Weisner, Dr.P.H. 


Senior Scientist, Alcohol Research Group, Western Consortium for Public Health; 
Adjunct Professor, Div. of Public Health Biology and Epidemiology, and Program 
Director/PI of NIAAA Program at School of Public Health, University of California at 
Berkeley; Adjunct Investigator, Evaluation of Alcohol & Drug Programs, Kaiser 
Foundation Research Institute; Member, Advisory Board of Alcohol & Drug Studies 
Program, University of California Extension: Editorial Board: Frontlines. Addiction 
Abstracts. International Research Monograph Series in the Addictions. Medical Care 
Research and Review: Member. Advisorv Committee. NIDA Center for Community- 
Based Treatment Research Methods, RTI; Panel Member, Treatment Effectiveness and 
Outcome, NIAAA; Member, Advisory Board, National Treatment Evaluation, ONDCP; 
Member, WHO Expert Advisory Panel on Drug Dependence & Alcohol Problems. 


Harry Wexler, Ph.D. 


Founder/Executive Director, The Psychology Center (NY/CA); PI, “Evaluation of Amity 
Prison Therapeutic Community for Substance Abusers,” NIDA/NDRI-NY/CA; 

S A/Consultant, Comprehensive Substance Abuse Programs for Correctional Populations, 
CSAT, (Illinois and South Carolina); Program Evaluator, “Comprehensive Residential 
Services for Substance-Abusing Women and Their Children,” CSAT/NDRI-NY/CA; Co- 
Guest Editor. Drugs & Society: Consulting Editor. Journal of Criminal Justice: Adjunct 
Associate Professor, Medical Psychology, Neuropsychiatric Institute, University of 
California. 
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Ken Winters, Ph.D. 


Senior Research Associate, Associate Professor, Dept, of Psychiatry, University of 
Minnesota; Instructor, “Assessment and Treatment of Adolescent Substance Abuse,” for 
Addiction Research Foundation, American Psychological Association, Johnson Institute, 
etc.; PI, NIDA; PI, Walker Foundation; Investigator, CSAP; Investigator, NIDA; 
Reviewer: Addiction, Alcoholism-Clinical and Experimental Research, Drug & Alcohol 
Dependence, Journal of Child & Adolescent Substance Abuse, Journal of Studies on 
Alcohol, Psychology of Addictive Behaviors; CSAT, NIDA. 


Eric Wish, Ph.D. 


Director, Center for Substance Abuse Research (CESAR), University of Maryland; 
Associate Professor, Dept, of Criminology & Criminal Justice, University of Maryland. 


Gary Zarkin 


Program Director and Senior Economist, Center for Economics Research, Health and 
Human Resource Economics, Research Triangle Institute . 
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This bibliography focuses on the unique aspects of evaluating on a systems level and is broken 

up into subject headings illustrating those aspects. 

Client-Treatment Matching 

Anglin, M. D., Hser, Y. I., & McGlothlin W. H. (1987). Sex differences in addict careers. 2. 
Becoming addicted. American Journal of Drug and Alcohol Abuse, 7i(l -2), 59-71 . 
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Treatment. American Journal of Drug and Alcohol Abuse, 7i(3), 253-80. 

Anglin, M. D., Ryan, T. M., Booth, M. W. , & Hser, Y. I. (1988, February). Ethnic differences 
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cXxQnXs. International Journal of the Addictions, 23(2), 125-49. 

Anglin, M. D., Booth M. W., Ryan, T. M., & Hser, Y. I. (1988, October). Ethnic differences in 
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International Congress on Alcoholism and Drug Dependence. 
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